-*- coding: utf-8 -*-

* Calibre OPDS client
** What's this?
This is a calibre plugin that is an OPDS client intended to read the contents of another calibre installation, find the differences to the current calibre and offer to copy books from the other calibre into the current calibre
** How do I install it?
Requires git and calibre installed:
 - Clone the repository:
   #+BEGIN_EXAMPLE
     git clone https://github.com/steinarb/opds-reader.git
   #+END_EXAMPLE
 - Install the plugin in calibre
   #+BEGIN_EXAMPLE
     cd opds-reader/calibre_plugin/
     calibre-customize -b .
   #+END_EXAMPLE
 - Start calibre (if calibre was already running, stop calibre and start it again)
 - Click the button "Preferences"
 - In the dialog "calibre - Preferences":
   - Under "Interface", click on the button "Toolbar"
   - In the dialog "calibre - Preferences - Toolbar":
     - In the dropdown, select "The main toolbar"
     - In "Available actions" scroll down to find "OPDS Client" and select it
     - Click the top arrow button (arrow pointing right)
     - Click the "Apply" button
   - Click the "Close" button
** How do I use it?
I made this tool to backup my book collection between two PCs in my home LAN, and that is the procedure I will document here:
  1. In the calibre you wish to copy from (in this example called calibre1.home.lan):
     1. Click Preferences
     2. In the "calibre - Preferences" dialog:
        1. Click "Sharing over the net"
        2. In the "calibre - Preferences - Sharing over the net" dialog:
           1. Click the "Start Server" button
           2. Select the checkbox "Run server automatically when calibre starts"
           3. Click the "Apply" button
        3. Click the "close" button
  2. In the calibre you wish to copy to
     1. Install this plugin (see the "How do I install it?" section)
     2. Click the "OPDS client" button
     3. In the "OPDS client" dialog
        1. Edit the "OPDS URL" value, change
           : http://localhost:8080/opds
           to
           : http://calibre1.home.lan:8080/opds
           and then press the RETURN key on the keyboard
        2. Click the "Download OPDS" button
        3. Wait until the OPDS feed has finished loading (this may take some time if there is a large number of books to load)
           - Note: if no books appear, try unchecking the "Hide books already in the library" checkbox.  If that makes a lot of books appear, it means that the two calibre instances have the same books
        4. select the books you wish to copy into the current calibre and click the "Download selected books"
           - calibre will start downloading and installing the books:
             - The Jobs counter in calibre's lower right corner, will show a decrementing number and the icon will spin
             - The book list will be updated as the books are downloaded
        5. The downloaded books will be in approximately the same order as in the original, but the time stamp will be the download time. To fix the time stamp, click on the "Fix timestamps of the selection" button
           - The updated timestamps may not show up immediatly, but they will show up after the first update of the display, and the books will be ordered according to the timestamp after stopping and starting calibre
** License
This calibre plugin is copyright Steinar Bang, 2015, and licensed Under GPL version 3.

See the LICENSE file for more detail.
* List of TODO items [27/41]
** DONE Create an icon
- The plugin API seems to mandate one for a plugin that is to add an action to the GUI
- Need an icon that doesn't have any copyright limitations, it doesn't have to be pretty
** DONE Put a list of books into the dialog
 - <2015-08-23 søn 08:50> Will use a [[http://doc.qt.io/qt-5/qtableview.html][QTableView]]
   - import:
     #+BEGIN_SRC python
       from PyQt5.Qt import QTableView
     #+END_SRC
   - Tried using the derived class BookView, but this failed because the parent (OpdsDialog) was missing the field iactions
   - Will try filling the QtTableView with BooksModel which derives from [[http://doc.qt.io/qt-5/qabstracttablemodel.html][QAbstractTableModel]]
     #+BEGIN_SRC python
       from calibre.gui2.library.models import BooksModel
     #+END_SRC
   - The argument to BooksModel.setData will be a list of SearchResult instances
     #+BEGIN_SRC python
       from calibre.gui2.store.search_result import SearchResult
     #+END_SRC
   - SearchResult contains the following fields:
     - store_name
     - cover_url
     - cover_data
     - title
     - author
     - price
     - detail_item
     - drm
     - formats
     - downloads
       - dictionary
     - affilate
       - boolean
     - plugin_author
     - create_browser
   - SearchResult equality is determined in the following way:
     #+BEGIN_SRC python
       return self.title == other.title and self.author == other.author and self.store_name == other.store_name and self.formats == other.formats
     #+END_SRC
     - The "formats" part of the comparison may be an issue when comparing with the current database? Could be that a comparison that excludes formats may be needed? E.g. I may want to keep ORIGINAL_EPUB in the calibre where I did the conversion, but not bother copying it to other calibres
   - An example usage of creating a list of SearchResult is in calibre in MobileReadStore.deserialize_books
** DONE Read RSS from the plugin and put the resulting data into a datastructure compatible with the calibre db API
 - <2015-08-23 søn 10:17> Will try putting the resulting data into SearchResult initially (as mentioned in the previous TODO item)
 - <2015-08-23 søn 11:05> The feedparser is already present in calibre
   - import statement:
     #+BEGIN_SRC python
       from calibre.web.feeds.feedparser import parse
     #+END_SRC
 - <2015-08-23 søn 12:02> The BooksModel used in view.py is directly connected to the database, i.e. can't use that BooksModel
   - Instead created a new BooksModel patterned on the one in the mobileread store
   - Use a new data structure OpdsBook instead of SearchResult
** DONE Populate the list in the GUI with data read from the RSS
 - <2015-08-23 søn 10:18> Hopefully this will be as simple as calling BooksModel.setData
** DONE Make the list of books look a little better (resize the dialog to make room for everything)
** DONE Add a checkbox to filter out newspapers
** DONE Add a checkbox to filter out books already present in the library
** DONE Make the book datamodel be Metadata (add a field to hold the parsed OPDS structure), and parse all available metadata info
** DONE Add a download button to download the selected books
 - <2015-09-04 fre 21:44> Makes a list of all book download links with EPUB first if found and download the first URL in the list
** DONE Fix line height after updates
** DONE Move OPDS reading to the model and use the model refresh instead of setting a new model
** DONE Restore the authors to the OPDS book list
** DONE Make sure all of the books in the library are listed
 - <2015-09-05 lør 10:34> Calibre is set up by default to only deliver 30 items
 - <2015-09-05 lør 23:20> Following the "next" links of the feed until there are no more "next"
** DONE Give feedback on the number of OPDS books downloaded
 - <2015-09-06 søn 12:43> Perhaps update the list after each 30 book chunk has been added?
** DONE Get a display value back for "updated"
 - <2015-09-06 søn 12:28> The value is back, but now there is a value for the initial empty lines
** DONE Reverse the order of requested downloads
 - <2015-09-06 søn 08:12> The idea is that what's started first will finish first and that this will give the same book order in the two calibres
** DONE Set the date/time of the copied book to the date of the original
 - <2015-09-05 lør 23:08> The date returned by "Updated" was the same for all books and the date/time of the last change to the db of the remote calibre
   - Opened an issue to replace this with the Metadata.timestamp attribute of the book: https://bugs.launchpad.net/bugs/1492651
 - <2015-09-06 søn 09:25> Use the ajax.py REST API to download the metadata for the remote book
   - Endpoints are:
     - /ajax/book/{book_id}/{library_id=None}
       - Return the metadata of the book as a JSON dictionary.
       - Query parameters: ?category_urls=true&id_is_uuid=false&device_for_template=None
     - /ajax/books/{library_id=None}
       - Return the metadata for the books as a JSON dictionary.
       - Query parameters: ?ids=all&category_urls=true&id_is_uuid=false&device_for_template=None
     - /ajax/categories/{library_id=None}
       - Return the list of top-level categories as a list of dictionaries
       - Each category has the form:
         #+BEGIN_SRC json
           {
             "name": "Display Name",
             "url": "URL that gives the JSON object corresponding to all entries in this category",
             "icon": "URL to icon of this category",
             "is_category": "False for the All Books and Newest categories, True for everything else"
           }
         #+END_SRC
     - /ajax/category/{encoded_name}/{library_id=None}
       - Return a dictionary describing the category specified by name
       - Query parameters: ?num=100&offset=0&sort=name&sort_order=asc
       - Response:
         #+BEGIN_SRC json
           {
               "category_name": "Category display name",
               "base_url": "Base URL for this category",
               "total_num": "Total numberof items in this category",
               "offset": "The offset for the items returned in this result",
               "num": "The number of items returned in this result",
               "sort": "How the returned items are sorted",
               "sort_order": "asc or desc",
               "subcategories": "List of sub categories of this category.",
               "items": "List of items in this category"
           }
         #+END_SRC
     - /ajax/books_in/{encoded_category}/{encoded_item}/{library_id=None}
       - Return the books (as list of ids) present in the specified category
       - Query parameters: ?num=100&offset=0&sort=title&sort_order=asc&get_additional_fields=
     - /ajax/search/{library_id=None}
       - Return the books (as list of ids) matching the specified search query.
       - Query parameters: ?num=100&offset=0&sort=title&sort_order=asc&query=
 - <2015-09-06 søn 10:30> Tried using the API
   - http://edwards.hjemme.lan:8080/ajax/books/ returned a 404
   - Added an
     : Accept: application/json
     header to the GET, but still got a 404
 - <2015-09-14 ma 23:56> Timestamps are now received and placed correctly on the book Metadata objects from the OPDS feed, the downloaded metadata hasn't been updated yet
 - <2015-09-15 ti 22:47> Added a "Fix timestamps of selection" button that can be clicked after download
   - I would have preferred to do this automatically, but that would require a callback that could be called after download, and no such callback system exists
   - The "Fix timestamps of selection" button only works for books that have a single author, the db.find_identical_books() method returns nothing when the metadata has more than one author
** DONE Make the "Fix timestamps of selection button" work for books with more than one author
** DONE Format the book list with different color for alternate lines
** DONE Add licensing information (GPLv3) on the project level
** DONE Add licensing and copyright information to all files
** DONE Add recently used dropdown to the opds_url configuration
** DONE Make the OPDS parsing more robust (hardcoded to the default structure of calibre right now)
** DONE Put the OPDS feed combobox on the main dialog
** DONE Add a catalog selection combobox
 - <2015-09-20 sø 12:13> Non-editable, populated from the catalogs in the root catalog when downloading a new feed
** DONE Add search in downloaded results
 - <2015-09-26 lø 19:55> Solved, but not entirely satsified with the results:
   - Only the currently visible items are searchable
   - Only the search matches are shown, instead of scrolling to the matches while showing all
     - Probably the best that can be done, using just built-in Qt functionality?
** TODO Read OPDS feeds other than calibre
 - <2015-09-18 fr 19:01> Some examples
   - feedbooks:
     : http://www.feedbooks.com/books/top.atom?category=FBFIC028000&lang=en
     or
     : http://www.feedbooks.com/catalog.atom
   - Internet archive:
     : http://bookserver.archive.org/catalog/
   - Pragmatic bookshelf:
     : http://pragprog.com/magazines.opds
   - ManyBooks:
     : http://www.manybooks.net/opds/index.php
   - Project Gutenberg:
     : http://m.gutenberg.org/ebooks/?format=opds
   - O'Reilly:
     : http://opds.oreilly.com/opds/
   - Baen ebooks:
     : http://www.baenebooks.com/stanza.aspx
 - <2015-10-04 søn 10:57> Need to differentiate between "navigation feeds" and "aquisition feeds"
   - For the calibre OPDS feeds
     - The top level calibre feed is a navigation feed
     - The "Newest" feed on the second level, is an "aquisition feed"
     - The "Authors" feed on the second level, is a "navigation feed" with the entries consisting of links to each author with a surname starting with a particular letter
   - A "navigation feed", is identified by:
     : kind=navigation
     in the parameter to the type definition in the "type" attribute of a link element, e.g. like so:
     #+BEGIN_SRC xml
       <link rel="start"   
             href="/opds-catalogs/root.xml"
             type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
     #+END_SRC
   - An "aquisition feed", is identified by:
     : kind=aquisition
     in the parameter to the type definition in the "type" attribute of a link element, e.g. like so:
     #+BEGIN_SRC xml
       <link rel="self"    
              href="/opds-catalogs/unpopular.xml"
              type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
     #+END_SRC
   - Unfortunately, the OPDS feeds don't have a "kind" parameter to the OPDS feeds:
     - Newest (which links to an aquisition feed, where all entries are books):
       #+BEGIN_SRC xml
         <link type="application/atom+xml;type=feed;profile=opds-catalog"
               href="/opds/navcatalog/4f6e6577657374"/>
       #+END_SRC
     - Authors (which links to a navigation feeds, where all entries are new OPDS feds):
       #+BEGIN_SRC xml
         <link type="application/atom+xml;type=feed;profile=opds-catalog" 
               href="/opds/navcatalog/4e617574686f7273"/>
       #+END_SRC
     - In other words: nothing to distinguish the two in just the link to the feed
       - I.e. it will be necessary to handle all feeds the same way, and just recurse on the links that are feeds
 - <2015-10-04 søn 11:54> Trying to distinguish the entries in "authors" from the entries in "Newest"
   - "Authors" entry:
     #+BEGIN_SRC python
       {
           u'updated': u'2015-09-28T15:49:33+00:00',
           u'updated_parsed': time.struct_time(tm_year=2015, tm_mon=9, tm_mday=28, tm_hour=15, tm_min=49, tm_sec=33, tm_wday=0, tm_yday=271, tm_isdst=0),
           u'links': [
               {
                   'href': u'http://localhost:8080/opds/categorygroup/617574686f7273/5c',
                   'type': u'application/atom+xml;type=feed;profile=opds-catalog',
                   u'rel': u'alternate'
               }
           ],
           u'title': u'\\',
           u'summary': u'1 items',
           u'content': [
               {
                   u'base': u'http://localhost:8080/opds/navcatalog/4e617574686f7273',
                   u'type': u'text/plain',
                   u'value': u'1 items',
                   u'language': None}
           ],
           u'guidislink': True,
           u'title_detail': {
               u'base': u'http://localhost:8080/opds/navcatalog/4e617574686f7273',
               u'type': u'text/plain',
               u'value': u'\\',
               u'language': None},
           u'link': u'calibre:category-group:authors:\\',
           u'id': u'calibre:category-group:authors:\\'
       }
     #+END_SRC
   - "Newest" entry:
     #+BEGIN_SRC python
       {
           u'updated': u'2015-09-28T15:49:33+00:00',
           u'updated_parsed': time.struct_time(tm_year=2015, tm_mon=9, tm_mday=28, tm_hour=15, tm_min=49, tm_sec=33, tm_wday=0, tm_yday=271, tm_isdst=0),
           u'links': [
               {
                   'href': u'http://localhost:8080/get/epub/460',
                   'type': u'application/epub+zip',
                   'rel': u'http://opds-spec.org/acquisition'
               },
               {
                   'href': u'http://localhost:8080/get/cover/460',
                   'type': u'image/jpeg',
                   'rel': u'http://opds-spec.org/cover'
               },
               {
                   'href': u'http://localhost:8080/get/thumb/460',
                   'type': u'image/jpeg',
                   'rel': u'http://opds-spec.org/thumbnail'
               }
           ],
           u'author': u'Michael Scott Rohan & Allan J. Scott',
           u'title': u'The Ice King',
           u'summary': u'TAGS: Fiction,
           General,
           Science Fiction<br />\n<p class="description">A Viking temple. A Viking ship. Both preserved in the clinging,
           black mud of the North Yorkshire estuary. Press and TV watch over the archaeologists\' shoulders as past and present merge.  And while huge,
           death-cold creatures stalk and destroy through the blizzards of an eerily early winter,
           modern computer science and the dark night-knowledge of the old Norse gods disinter a terrible truth about a past that is sleeping,
           not dead.</p>',
           u'content': [
               {
                   u'base': u'http://localhost:8080/opds/navcatalog/4f6e6577657374',
                   u'type': u'application/xhtml+xml',
                   u'value': u'TAGS: Fiction,
                   General,
                   Science Fiction<br />\n<p class="description">A Viking temple. A Viking ship. Both preserved in the clinging,
                   black mud of the North Yorkshire estuary. Press and TV watch over the archaeologists\' shoulders as past and present merge.  And while huge,
                   death-cold creatures stalk and destroy through the blizzards of an eerily early winter,
                   modern computer science and the dark night-knowledge of the old Norse gods disinter a terrible truth about a past that is sleeping,
                   not dead.</p>',
                   u'language': None}
           ],
           u'guidislink': True,
           u'title_detail': {
               u'base': u'http://localhost:8080/opds/navcatalog/4f6e6577657374',
               u'type': u'text/plain',
               u'value': u'The Ice King',
               u'language': None},
           u'link': u'urn:uuid:c34a0ada-ffe2-4d51-a820-9a84b7664003',
           u'authors': [
               {
                   u'name': u'Michael Scott Rohan & Allan J. Scott'
               }
           ],
           u'author_detail': {
               u'name': u'Michael Scott Rohan & Allan J. Scott'
           },
           u'id': u'urn:uuid:c34a0ada-ffe2-4d51-a820-9a84b7664003'
       }
     #+END_SRC
** TODO Add auto discovery of calibre instances in the LAN
 - <2015-09-06 søn 11:49> Perhaps use the Bonjour protocol? (is this what FBReaderJ uses?)
** TODO Find out why the OPDS reader dialog sometimes disappear after downloading the OPDS
** TODO Add username/password information to saved opds_url values
** TODO Migrate own code from underscore separation to camelCase (Python has a camelCase modula/pascal feel to it)
** TODO Find out why some books (in PDF...?) aren't downloaded
** TODO Explore the documentation format to see if it is relevant to this plugin
** TODO Try to keep the line length correct during intermediate model updates
 - <2015-09-06 søn 16:10> When updating the book list model after each OPDS chunk, the line heights are wrong
   - They are corrected after the final read but they look a bit silly during the intermediate chunks
** TODO Get better matching with existing books (the "Maven cookbook" was already present, but it still showed up)
** DONE Add configuration options for defaults for the "hide" checkboxes
** TODO Remove all leftover debug trace
** TODO Copy read marks in calibre's reader from the remote
** TODO Refresh the list as books are downloaded (suppress downloaded books from the list)
** TODO Add cover thumbnails to the list of books
** TODO Add an exclusion list (a list of books that should be permanently hidden from the comparison)
